System and method for quantification of latent effects on user interactions with an online presence in a distributed computer network resulting from content distributed through a distinct content delivery network

ABSTRACT

Embodiments of the present disclosure relate to data correlation of data pertaining to content distributed through distinct content delivery networks including offline networks and data related to user interaction with an online presence on the distributed computer network and uses for such correlated data, including to measure and quantify latent effects of the content distributed through a distinct content delivery network, such as an offline network, on user&#39;s interactions with the online presence on the distributed computer network.

RELATED APPLICATION(S)

This application is a continuation of, and claims a benefit of priorityunder 35 U.S.C. 120 of the filing date of U.S. patent application Ser.No. 16/363,801, filed Mar. 25, 2019, issued as U.S. Pat. No. 11,132,706,entitled “SYSTEM AND METHOD FOR QUANTIFICATION OF LATENT EFFECTS ON USERINTERACTIONS WITH AN ONLINE PRESENCE IN A DISTRIBUTED COMPUTER NETWORKRESULTING FROM CONTENT DISTRIBUTED THROUGH A DISTINCT CONTENT DELIVERYNETWORK,” which claims a benefit of priority under 35 U.S.C. § 119(e)from the filing date of U.S. Provisional Application No. 62/648,155,filed on Mar. 26, 2018, entitled “SYSTEMS AND METHODS FOR QUANTIFICATIONOF LATENT EFFECTS ON USER INTERACTIONS WITH AN ONLINE PRESENCE IN ADISTRIBUTED COMPUTER NETWORK RESULTING FROM CONTENT DISTRIBUTED THROUGHA DISTINCT CONTENT DELIVERY NETWORK,” by Swinson, and U.S. ProvisionalApplication No. 62/781,336, filed on Dec. 18, 2018, entitled “SYSTEMSAND METHODS FOR CORRELATION OF USER INTERACTIONS WITH AN ONLINE PRESENCEIN A DISTRIBUTED COMPUTER NETWORK AND CONTENT DISTRIBUTED THROUGH ADISTINCT CONTENT DELIVERY NETWORK AND USES FOR SAME, INCLUDINGQUANTIFICATION OF LATENT EFFECTS ON SUCH USER INTERACTIONS by Swinsonand Driscoll. The entire contents of each application referenced in thisparagraph are fully incorporated by reference herein in their entiretyfor all purposes.

TECHNICAL FIELD

The present disclosure relates generally to distributed and networkedcontent delivery networks. More particularly, embodiments of the presentdisclosure relate to the measurement and quantification of userinteraction with an online presence, such as a website, on a distributedcomputer network based on content distributed through a distinct contentdelivery network. Even more specifically, embodiments of the presentdisclosure relate to the measurement and quantification of the latenteffects of content distributed through the distinct content deliverynetworks on user interaction with the online presence on the distributedcomputer network.

BACKGROUND

With the advent of the Internet, many aspects of modern life are nowdigitally connected through the seemingly ubiquitous smart phones, smarttelevisions (TV), smart home appliances, Internet of Things (IoT)devices, websites, mobile apps, etc. Even so, many more analog aspectsremain disconnected from this digital world. Linear TV is an example ofan offline medium that is disconnected from the digital world.

“Linear TV” refers to real time (live) television services that transmitTV program schedules. Almost all broadcast TV services can be consideredas linear TV. Non-linear TV covers streamlining and on-demandprogramming, which can be viewed at any time and is not constrained byreal-time broadcast schedules. Video-on-demand (VOD) and nearvideo-on-demand (NVOD) transmissions of pay-per-view programs overchannel feeds are examples of non-linear TV.

Because forms of TV are an offline medium, it may be difficult toautomatically collect information on some viewers of TV. This creates adata gap problem. To address this data gap program, Nielsen MediaResearch devised audience measurement systems to determine the audiencesize and composition of television programming in the United States.Nielsen television ratings are gathered in one of two ways—using viewerdiaries or set meters attached to TVs in selected homes. The formerrequires a target audience self-record their viewing habits. The latterrequires a special device to collect specific viewing habits on a minuteto minute basis and send the collected information to Nielsen's systemover a phone line.

While Nielsen's audience measurement systems can provide some quantifiedmeasures of audience response to TV programs, the Nielsen televisionratings do not measure conversion rates for TV commercials (referred toherein as “creatines” or “spots”), either from linear TV or non-linearTV. This is, in part, because there is a natural distinction between twomediums: online (e.g., search engine marketing) and offline (e.g.,linear or non-linear TV). The online medium is effective when consumersare already accessing the Internet through a website or a mobile app.When a user is attracted to a product and visits a website for theproduct (e.g., when attracted by an advertisement on a website), thereis a session associated with that advertising channel. Thus, whethersuch a session results in a sale (or conversion) is a relativelystraightforward process that can be done through tracking the session atthe website (e.g., using a tracking pixel embedded in a page or pages ofthe website that sends data to a server of the website operator or athird-party entity).

The offline medium, on the other hand, aims to drive consumers first tothe Internet and then to the product's website or app. Unlike the onlinemedium, there is neither session tracking nor a direct relationshipbetween the offline medium and the desired result. Thus, suppose a spotthat aired on linear TV encouraged its viewer to visit a website ordownload an app, it can be extremely difficult to measure the impact ofthat spot and quantifiably attribute any website visit or app downloadto the particular spot. The natural distinction between the two mediumscreates a data gap between them.

From the perspective of a website, although the general trafficcontribution from TV to the website can be assessed from the immediatelift approximation (e.g., after a spot aired on TV at a certain time,there is an immediate visitor lift at the website), there is not a cleartag for the sales contribution from TV. That is, each online sessiondoes not have a tag “from TV,” because there is no one-to-onerelationship between a website visitor and TV viewership. This creates achallenge to estimate TV contribution to website conversions.

SUMMARY

Embodiments of the present disclosure thus relate to data correlation ofdata pertaining to content distributed through distinct content deliverynetworks including offline networks and data related to user interactionwith an online presence on the distributed computer network and uses forsuch correlated data, including to measure and quantify latent effectsof the distributed content on user's interactions with the onlinepresence on the distributed computer network.

Specifically, embodiments of a quantification system may estimate thelatency factor of a retailer's creatives. In particular, certainembodiments of quantification systems may leverage rich data sets (TVviewing and retailer's online access data (e.g., app install data) andstatistics), to determine a latency factor to help retailers andassociated creators of creatives better understand the efficiency orefficacy of their advertising.

In particular, data of a quantification system can be utilized for avariety of purposes and ends by quantification systems. As but oneexample, this data can be used by the quantification system to calculateone or more metrics associated with both a creative, or group ofcreatives, aired by the retailer on the offline network and theretailer's presence on the online network. These metrics may includeefficiency, relative performance, lift or response, conversion rate,spend per conversion, or other metrics as discussed. These metrics, ordata determined therefrom, can be displayed to a user accessing thequantification system in an intuitive interface to allow the user toassess the efficacy or other insights into the retailer's creatines.

It is thus desired to quantify metrics using a latency factor which mayaccount for the effects of the airing of a creative that occur outsideof some baseline window (e.g., time period, such as 5 minutes, 10minutes, 30 minutes, etc.) following the original airing of thecreative. Such a latency factor may be associated with, or otherwisereflect, an expected response to an airing of a spot outside of thebassline window (e.g., relative to a response within the baselinewindow) vis a vis the retailer's online presence. Thus, a latency factorcan be applied to almost any other determined metric to estimate oradjust that metric to account for user's interactions that occur outsideof the baseline window and within a selected time period (e.g., 30 days,60 days, a quarter, a year, etc.).

By determining such a latency factor, users can access and determine thepotential response from a creative more quickly, improving the computerperformance by allowing metrics and other data to be determined morequickly and interfaces to be generated more quickly. It also can providethe capability of measuring delayed results on a granular level. Otherapproaches either will not be able to gauge the full response to anindividual spot, or will only be able to estimate the full response onan aggregated level. Such a latency factor may be usefully applied to anumber of metrics that may be determined by a quantification system,including for example traffic lift or conversion.

In one embodiment, a quantification system may include a data storestoring spot viewing data comprising data on user views of a first spotassociated with an entity through an offline network, and userinteraction event data comprising data on user interactions with anonline presence on an online network, wherein the online presence isassociated with the entity associated with the first spot. Thequantification system may associate one or more user views of the firstspot through the offline network with corresponding user interactionswith the online presence of the entity on the online network. A firstset of user views of the first spot that are associated withcorresponding user interactions with the online presence on the onlinenetwork may be determined and a first conditional probability curvedetermined for the first set of views. A latency factor can bedetermined based on a first portion of the first conditional probabilitycurve corresponding to a time at a baseline time window and a secondportion of the first conditional probability curve outside the baselinetime window, wherein the latency factor reflects an expected response atthe entity's online presence on the online network and wherein theresponse is a response to the spot airing on the offline network outsidethat time window. A metric associated with the online presence can beadjusted based on the latency factor and an interface generated based onthe adjusted metric.

In certain embodiments, the spot viewing data comprises data on userviews of a second spot, and the quantification system may associate oneor more user views of the second spot through the offline network withcorresponding user interactions with the online presence of the entityon the online network, wherein the second spot aired at a similar timeto the first spot. A second set of user views of the second spot thatare associated with corresponding user interactions with the onlinepresence on the online network can be determined and a secondconditional probability curve determined for the second set of views. Acumulative probability difference curve can be determined based on thefirst conditional probability curve and the second conditionalprobability curve. In this embodiment, determining the latency factorbased on the first portion of the first conditional probability curvecorresponding to the baseline time window and the second portion of thefirst conditional probability curve outside the baseline time windowcomprises determining a first portion of the cumulative probabilitydifference curve corresponding to the baseline time window and a secondportion of the cumulative probability difference curve outside thebaseline time window.

In some embodiments, the data store of the quantification system storesspot airing data comprising data on when the spot associated with theentity was aired on the offline network, and the spot airing data isenhanced by adjusting a scheduled start time of at least one spot airingof the spot airing data.

This enhancement may be done, in one embodiment, by associating eachview of the spot in the spot viewing data with a corresponding instanceof the airing of the spot in the spot airing data based on time toadjust a spot viewing time of the view of the spot. The association canbe done, for example using IP address associated with both a user viewof the spot and the corresponding user interaction with the onlinepresence.

These, and other, aspects of the disclosure will be better appreciatedand understood when considered in conjunction with the followingdescription and the accompanying drawings. It should be understood,however, that the following description, while indicating variousembodiments of the disclosure and numerous specific details thereof, isgiven by way of illustration and not of limitation. Many substitutions,modifications, additions and/or rearrangements may be made within thescope of the disclosure without departing from the spirit thereof, andthe disclosure includes all such substitutions, modifications, additionsand/or rearrangements.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings accompanying and forming part of this specification areincluded to depict certain aspects of the invention. A clearerimpression of the invention, and of the components and operation ofsystems provided with the invention, will become more readily apparentby referring to the exemplary, and therefore nonlimiting, embodimentsillustrated in the drawings, wherein identical reference numeralsdesignate the same components. Note that the features illustrated in thedrawings are not necessarily drawn to scale.

FIG. 1A is a diagram of one embodiment of a topology for aquantification system.

FIG. 1B is a diagram of one embodiment of networked environment for aquantification system.

FIGS. 2A and 2B are a diagram of one embodiment of an interface forpresenting response to creatines.

FIG. 3 is a block diagram for an example utilized in the disclosure.

FIG. 4 is a flow diagram for one embodiment of a determining a latencyfactor.

FIG. 5 is a diagram graphically depicting one embodiment of determininga latency factor.

FIG. 6 is a diagram of one embodiment of an interface.

FIG. 7 is a diagram of one embodiment of an interface.

FIG. 8 is a diagram of one embodiment of a representation of a histogramassociated with scheduled start time adjustment.

FIG. 9 is a flow diagram for one embodiment of a method for using alatency factor to quantify a metric.

FIG. 10 is a graph of an example latency profile.

FIG. 11 is a diagram graphically depicting an example utilized in thedisclosure.

DETAILED DESCRIPTION

The invention and the various features and advantageous details thereofare explained more fully with reference to the nonlimiting embodimentsthat are illustrated in the accompanying drawings and detailed in thefollowing description. Descriptions of well-known starting materials,processing techniques, components and equipment are omitted so as not tounnecessarily obscure the invention in detail. It should be understood,however, that the detailed description and the specific examples, whileindicating some embodiments of the invention, are given by way ofillustration only and not by way of limitation. Various substitutions,modifications, additions and/or rearrangements within the spirit and/orscope of the underlying inventive concept will become apparent to thoseskilled in the art from this disclosure.

Before delving into more detail regarding the specific embodimentsdisclosed herein, some context may be helpful. In the currentenvironment for content delivery and consumption, there is a naturaldistinction between at least two mediums or networks. Today, there isthe distribution network generally referred to as the “online” networkfor content distribution, where content is delivered over a computerbased network such the Internet World Wide Web (WWW) or the like. Thereare also a number of what are considered “offline” networks for contentdistribution. These offline networks are traditional distributionnetworks such as television (TV) (whether transmitted, cable or networkbased) or radio.

There is a great deal of crossover between these types of distributionnetworks, both from a technology standpoint and from a businessperspective. For example, many retailers, sellers, manufacturers orother entities (collectively retailers) of products or services(collectively products) have presences on the online distributionnetwork (e.g., websites on the WWW, application to install on a device,or the like), and in some cases, may only have a public facing presenceon the online content distribution network (e.g., may only have awebsite on the WWW, not any brick and mortar retail locations). Theseretailers may, however, advertise their products, sites applications,etc. both through an online distribution network (e.g., through searchengine marketing or banner ads) or through offline distribution network(e.g., traditional TV advertisements).

Advertising through an online medium is effective when consumers arealready accessing the Internet via a website or mobile application. Theefficacy of such advertisements is easily assessed. When a user isattracted to the retailer's website through an advertisement in anonline medium, there is a session associated with that advertisingchannel, so whether such session results in sale (or conversion) isrelatively straightforward.

Advertisements of a retailer distributed through the offline medium, onthe other hand, aim to drive consumers or other users first to theonline network (e.g., Internet or WWW) and then to the retailer's siteor application download. However, because of the technical impedimentsposed by the general separation in direct connectivity between theonline distribution network and the offline distribution network, theassessment of the efficacy of advertisements distributed through theoffline medium on the interaction of users with the retailer's onlinesite or application may be extremely difficult to assess.

Embodiments of systems and methods as disclosed herein may determinecertain metrics such as user views, application installs, conversions(e.g., conversion events such as retail sales or application installs)or other types of data with respect to user's interaction with aretailer's online presence that occur as a result of advertisements inthe offline network. It should be noted here, that while embodimentsherein are described in association with a “retailer”, the term“retailer” is used purely for the sake of convince, and embodiments willapply equally well to any other entity or organization that may operatean online presence and engage in presentation of content such asadvertising through an offline network without loss of generality andthe term retailer as used herein and throughout will be taken to referto all such organizations or entities.

Referring to FIG. 1A, embodiments of the systems and methods forquantification of these metrics may include one or more servers coupledto the online network (e.g., the Internet, an internet, a local areanetwork (LAN), a wide area network (WAN), or the like). These systemscan thus receive, collect or otherwise obtain a wide variety of datathrough the online network. This data may include data on a retailer'ssite such as event data associated with user views of the retailer'swebsite, application installs, product purchases, time spent on theretailer's site or a wide variety of other data (including analyticsdata) associated with a user interaction with the retailer's site (orapplication, which collectively will be understood to be referred towhen a retailer's site or online presence is discussed). Such data isusually associated with an Internet Protocol (IP) address of the user'sdevice used to access the retailer's online presence.

Additionally, data that may be obtained at such quantification systemsmay include data on the offline network including user interactions withthe offline network and advertising presented through the offlinenetwork. Typically, each individual advertisement is referred to as a“spot” or a “creative”. Thus, the data that can be obtained may comefrom the various channels or TV networks (e.g., ABC, CBS, NBC, HGTV,ESPN, Bravo, etc.). This data may include data on programming (e.g.,shows and times of airing in various markets), program demographics,times creatives were aired, pricing of air time, or the like. Certaindevice manufacturers of connectivity providers (e.g., internet serviceproviders or cable providers) may also obtain or collect data on userinteractions with the offline network or programs or creatives airingthereon. For example, many “smart” TVs, digital video recorders (DVRs)(e.g., TiVos) or the like may be connected both to the offline networkand the online network. These devices may thus collect data on user'sviewing, including what programs or TV channels were watched at whattimes, what programs were recorded, what programs (or portions thereof)were fast forwarded through or other data. This data can be reportedback to the manufacturers or connectivity providers who may, in turn,provide such data to the quantification system. Moreover, because suchdata has been reported from a “smart” device that has connectivity tothe online network, the IP address of the user on whom such data wascollected may also be reported or associated with such data.

The quantification system may thus obtain these types (or a wide varietyof other types) of data, on user's interactions with the offline network(e.g., what they saw or watched and when), what programs were aired onwhat channels at what time, what creatives were aired on what channelsand at what time, who (e.g., what IP addresses) saw those creatives, whovisited the retailer's online presence, who made a purchase, installedan application, etc., from the retailer's online presence or other data.

This data may be enhanced by the quantification system such that thequantification system may determine and store a set of enhanced datawhereby the user's interactions with the offline network (e.g., theircontent viewing events in the offline network) may be associated withtheir interaction events with the online network, including a retailer's(e.g., associated with one or more creatives) presence in the onlinenetwork. This enhancement may occur using the IP addresses of the eventdata from the retailer's online presence and the IP address of theuser's interactions with the offline network (e.g., as determined fromthe user's “smart” device used to access the offline network) such thatthe user's online interactions can be correlated or otherwise associatedwith the user's interaction with the offline network.

The quantification system can thus determine which users viewed whatprograms and creatives and subsequently accessed the retailer's onlinesites, made purchases (including app installs) from the retailer'sonline presence or other data. This data can be used by thequantification system to calculate one or more metrics associated withboth a creative or group of creatives aired by the retailer on theoffline network and the retailer's presence on the online network. Thesemetrics may include efficiency, relative performance, response,conversion rate, spend per conversion, or other metrics. These metrics,or data determined therefrom, can be displayed to a user accessing thequantification system in an intuitive interface to allow the user toassess the efficacy or other insights into the retailer's creatines.

It may be useful here to have an understanding of embodiments toillustrate an example distributed and networked environment that includean embodiment of such a quantification system. Moving to FIG. 1B, asdiscussed, offline content providers may provide content (e.g.,television programs, movies, videos, audio or music content, etc.) toviewers at their devices 110 (e.g., televisions, radios or other viewingor listening devices) over an offline network 104. An offline networks104 may usually be a traditional distribution networks such astelevision (TV) (whether transmitted, cable or computer network based)or radio (e.g., again, transmitted, satellite or computer networkbased). It will be understood, however, that though network 104 isreferred to as “offline” network and may be a traditional distributionnetwork, the “offline network” or, portions thereof, may also be, orinclude, an online network (e.g., the Internet, an internet, a localarea network (LAN), a wide area network (WAN), or the like), as will beunderstood. For example, offline content providers 120 may includecontent providers such as Hulu, Amazon, YouTube, Netflix or other“streaming services” that provide content over an online network thatmay also be, or has typically been, provided over a traditionaldistribution network.

For example, offline data can be obtained from or provided by TVnetworks. Prelogs and postlogs are TV network airing logs. “Prelog”refers to the planned schedule of TV spot purchases for the client bynetwork, date, and time that is identified in advance of the airing.“Postlog” refers to the actual times when spots aired on TV networks.Rates are how much the TV networks charge for commercial spots placed atvarious times of the day. Likewise, spot airing data, program schedules,and program demographics can be obtained or provided by TV networks ormedia agencies.

Online data can be obtained from or provided by data providers. Thesecan include clickstream data from a data analytics provider which caninclude the number of unique visitors (UVs) for a website, the number ofapplications (apps) downloaded from an electronic site or store, thenumber of purchases made on a website or app (e.g., conversions), etc.Online data can be collected from a variety of sources, includingwebsites, digital devices, consumer electronics, etc.

The offline data and online data collected at the quantification systemcan be processed so that they are merged temporally or in another mannerto determine an account (e.g., minute-by-minute account of what isoccurring in both the offline world and the online world. As an example,a graph generated based on TV viewership (offline data) can be overlaidon top of another graph generated based on UVs for a website (onlinedata) and presented to a user over the Internet through a user interfacerunning on the user's device.

It may now be useful to describe the operation of such quantificationsystems 170 and the networked environment in which they operate in moredetail. Devices 110 may collect viewing data on what content the usersof those devices 110 are viewing. As noted, certain device manufacturersor connectivity providers (e.g., internet service providers or cableproviders) may obtain or collect user viewing data on user interactionswith the content (e.g., programs or creatines airing thereon) providedby the offline content provider 120. For example, many “smart” TVs,digital video recorders (DVRs) (e.g., TiVos) or the like may beconnected both to the offline network 104 and an online network 102.These devices 110 may thus collect viewing data on user's viewing,including what programs or TV channels were watched at what times, whatprograms were recorded, what programs (or portions thereof) were fastforwarded through or other data.

This viewing data can be reported back to the smart device provider 130providers who may, in turn, provide such data to the quantificationsystem 170 (e.g., over online network 102). Moreover, because such datahas been reported from a “smart” device that has connectivity to theonline network 102, the IP address of the user on whom such data wascollected may also be reported or associated with such data.

Specifically, in one embodiment, a smart device 110 may fingerprint(e.g., hash) the content being displayed on the screen of the device 110and report these fingerprints, an associated time of viewing, anidentifier for the device 110, an IP address associated with the deviceon online network 102 or other viewing data to the smart device provider130. At some point quantification system 170 may provide the spots beingaired by offline content providers 120 (e.g., the actual content ofthose spots) to smart device provider 130. The smart device provider 130may thus fingerprint these provided spots in the same or a similarmanner to the fingerprints generated by the devices 110.

Accordingly, when the viewing data is received by the smart deviceprovider 130 from the devices 110, the smart device provider may be ableto determine from the fingerprints of the viewing data and theassociated time of viewing (or timestamp) which spot was viewed on whatdevice at what time. This spot viewing data 172 (e.g., an identifier ofthe spot was viewed, the time of viewing, a start time of the content(e.g., program) in which the spot was included, the identifier of thedevice, an IP address for the device, or other spot viewing data) may beprovided by the smart device provider 130 at some interval to thequantification system 170 (or the quantification system 170 may requestand receive such data, etc.) where it is stored at the quantificationsystem 170.

Specifically, in one embodiment, spot viewing data may include contentdata or IP that contains details about content (e.g., network, show, ortime) that devices 110 are displaying. Delayed viewing may be capturedby comparing Scheduled Start Time and Content Recognition Start Time. Acommercial IP that may contains details about which creatines (e.g., aspot identifier, or spot_id) were detected on devices 110 at what time,and during which content. An IP age may contain details about which IPaddresses were in use by which device 110, and during what time period.Other data is possible and is fully contemplated herein.

Moreover, in certain embodiments, the quantification system 170 mayinitially enhance such spot viewing data 172 when it is received. Forexample, the spot viewing data 172 may be enhanced as follows: where#delay in the following example, refers to the difference betweenScheduled Start Time and Content Recognition Time:

df[′recognition_delay′]   = (df_recognition_start_time −df.content_start_media_time_millis)   − df.scheduled_start_time; df[′delay_min′] = (df.recognition_delay.astype(′timedelta64[m])Here, spot_recognition_start_time-spot detection_start_millis gives theapproximate actual spot start time and

df[′spot_start_time′] = (df.spot_recognition_start_time −df.spot_detection_start_millis.astype(′timedelta64[ms]′)should (e.g., approximately) match spot.dt for both delayed and liveviews and

df[′scheduled_spot_time′] = df.spot_start_time − df.recognition_delay

As may be realized, for a variety of reasons this spot viewing data 172may be somewhat incongruent with the spots as provided in associationwith the content provided by the offline content provider 120. One ofthe biggest problems is that the timing of the viewing of a spot may becompletely divorced from the time the spot was originally aired (e.g.,broadcast over the air or a cable network by the offline contentprovider 120). In particular, the increasing prevalence of digital videorecorders (DVRs such as Tivos or the like) has allowed content to beviewed at almost any time desired, and the viewing of that content to bepaused or otherwise manipulated during viewing. Moreover, the timing ofthe viewing of the spot may not correlate with the airing time of thespot because of other factor such as time zone differences or biases orthe inability to correctly identify a spot or a wide variety of otherreasons.

It is thus desirable for a variety of reasons to enhance or adjust thespot viewing data 172 as reported from the smart device provider 130 todetermine that the spot viewing data is associated with the correct timeat which the spot actually aired (e.g., by offline content provider 120)and is associated with the correct spot.

Accordingly, in one embodiment, quantification system 170 may alsoreceive spot airing data 174 from spot airing data provider 140 over theonline network 102. This spot airing data 174 may be a file identifyinga set of instances of spot airings, where each instance includes anidentifier for a spot, a time of airing, and a network (or other offlinecontent provider identifier) on which the spot was aired. Spot airingdata provider 140 may be, for example, a market research firm such asKantar or the like.

Thus, once the spot viewing data 172 and the spot airing data 174 arereceived or obtained by the quantification system 170. The spot viewingdata may be enhanced or adjusted using the spot airing data 174 toobtain the enhanced and adjusted viewing data 178. In one embodiment,the scheduled start time (e.g., associated with the spot viewing data)may first be adjusted (STEP 182). Such an analysis may be performed overa sets of the spot viewing data 172 where a discrepancy has beendetermined.

The first step here may be to aggregate the spot viewing data at asegment level. This may be aggregating spot viewing data at a networklevel or geographic level. All data is grouped together to create ahistogram by delay time in relation to a time when content was supposedto have aired. So such a histogram is number of users (y axis) vs. delaytime (e.g., x axis). If a periodicity in spikes is determined thesespikes can be analyzed at more granular level (e.g., by geography). Oncenetworks or geographies have been identified a scheduled start time canbe adjusted to time shift particular period of times “back to zero”. Bycorrecting the scheduled start time of the content which was beingviewed, this may allow similar shifting of the actual airing time of thespot so that it can be better aligned or matched with the spot airingdata 174 as will be discussed in more detail.

With reference briefly to FIG. 8 , in one embodiment, due to incorrectgeographic data (or other causes) it is difficult to adjust easily theScheduled Start Time from East Coast to Local Time (which is reflectedin Content Recognition Start Time). From the examples in FIG. 8 it canbe seen that the difference between Content Recognition and ScheduledStart Times (e.g., delay minutes or delay_min) tends to follow cleanhourly or half-hourly intervals (due to geographic location relative toEast Coast), though certain network may be different.

Thus, in one embodiment, to determine if a scheduled start time shouldbe adjusted, the quantification system 170 may loop through allnetworks, and all delay_minutes that are a multiple of 30 in spotviewing data 172. The relative size of the initial minute to each of the30-minute intervals for each network may be determined and if thedelayed spike is more than 2% of the initial minute, the Scheduled StartTime of associated instances of the spot viewing data may be adjusted byadding delay_min for data in the 5 minute interval [delay_minute,delay_minute+5).

In certain embodiments, there may be networks or channels where allviewership is delayed (i.e., initial minute size==0), so the relativesize of the delayed spike is ∞. All of these delayed spikes may beincluded in the adjustment.

This scheduled start time adjustment may have the effect of pullingthose delayed spikes back to the origin, where they would have been ifcorrectly identified as live views. It may also have the effect ofpulling back a small amount of actually delayed viewership as if it werelive.

Accordingly, such scheduled start time adjustment may enhance the spotviewing data 172 through the creation of new fields, where correctedstart time is the adjusted Scheduled Start Time:

df.loc[:, ′corrected_delay′]  = (df.content_recognition_start_time)  −df.content_start_media_time_millis) − df.corrected_start_time df.loc[:,′corrected_delay_min′] = df.corrected_delay.astype(′timedelta64[m]′)

Returning then to FIG. 1B, the spot viewing data 172 may also bede-duplicated (STEP 184). It may be understood that the fingerprintingdone by the devices 110 or smart device provider 130 may not be perfect.Moreover, often times enterprises (e.g., retailers or the like) may airspots to test different (but very similar) creatines. Thus, thefingerprint may misidentify the spots watched on devices 110.Accordingly, spot viewing data 172 may include duplicative entries.Based on the spot airing data 174 the instance of the spot viewing data172 corresponding to the actual spot aired on the network at that timemay be identified using the spot airing data 174 and the duplicativeentries removed or merged with the identified correct entry.

Specifically, instances of spot viewing data 172 corresponding to thesame device 110 viewing a spot (e.g., on the same network or inassociation with the same content being viewed) at the same time (orwithin some tolerance of the same time), may be de-duplicated bydetermining the actual spot that was airing on that network at the timefrom the spot airing data 174 and merging or removing the duplicativeentries. It will be noted, that this de-duplication (STEP 184) may beperformed as part of the joining of spot viewing data 172 and spotairing data 174 as discussed below.

Network adjustment of the networks of the spot viewing data 172 may alsobe performed (STEP 186). In some cases, the smart device provider 130may only track certain data from certain geographic areas (e.g., the top50 designated market areas (DMAs)) and only for certain offline contentproviders 120 (e.g., certain channels such as the top 300). Accordingly,if spot viewing data 172 can't be matched to a particular network it maybe determined that the instance of a viewing of a spot may correspond toa view of that spot when the spot was aired (e.g., is a “live” viewingof the spot).

In this manner, based on the identity of the spot it can be matched to acorresponding instance of that spot airing at that time in spot airingdata 174 (e.g., based on the identifier of the spot) and thecorresponding network on which the spot aired assigned to that instanceof the spot viewing data 172. Additionally, mapping between localaffiliates and parent networks may also be performed at this step.Certain channels as included in the spot viewing data 172 may beidentify by local call signs (e.g., KXAN) and based on a database, thatcorrelates local affiliates with nationwide networks, the propernationwide network (e.g., ABC, CBS, NBC, etc.) assigned to instance ofspot viewing data 172.

Each instance of the spot viewing data 172 can then be identified with acorresponding instance of the airing of a spot as included in the spotairing data 174. Such a merging may be done by performing a join betweenthe instance of the spot viewing data 172 and the spot airing data 174based on a network associated with each instance and a spot identifier.The closest matches may be kept where the spot viewing time within thespot viewing data 172 and the spot airing data are within some tolerance(e.g., 15 seconds or the like).

Accordingly, the spot viewing data 172 may be enhanced or adjusted andthis enhanced or adjusted viewing data 178 stored at the quantificationsystem 170. Similarly, the user interaction event data 176 received atthe quantification system 170 may likewise be enhanced or adjusted. Thisuser interaction event data 176 may include data on a retailer's sitesuch as event data associated with user views of the retailer's website,application installs, product purchases, time spent on the retailer'ssite or a wide variety of other data (including analytics data)associated with a user interaction with the retailer's site (orapplication, which collectively will be understood to be referred towhen a retailer's site or online presence is discussed). Such data isusually associated with an IP address of the user's device used toaccess the retailer's online presence and a time stamp or time window.

Initially, this user interaction event data 176 may be IP filtered (STEP188). This IP filtering may include the removal of event data 176associated with any IP address known to be a bad or malicious IP addressor that has a volume of traffic that may indicate that the IP address isa bot or is associated with development or testing of the retailer'ssite. Multiple sessions or events associated with a similar IP addressor subnets of an IP address may also be excluded or consolidated.

The user interaction event data 176 may also be sessionized (STEP 190).As discussed, the user events in the user interaction event data 176 mayhave associated time stamps. This data may be, for example, clickstreamdata with associated IP addresses and one or more events.

Accordingly these events may be grouped into sessions by thequantification system 170. The sessions into which the event data 176may be grouped may be different than sessions as defined by theretailers or other entities and may utilize different criteria orcriteria specific to the quantification system 170. This sessionizationmay be based on one or more criteria, where these criteria may berelated to the timing or proximity of timing of the events in the eventdata 176. Moreover, the set of criteria for a session may differ basedupon the associated retailer site 150 on which the events 176 werecollected. For example, a large retail site may have a short sessionperiod

Accordingly, the enhanced and adjusted user interaction event data 192may be grouped into sessions. Thus, for each retailer's site 150 theremay be a set of associated sessions, each session comprising one or moreuser interaction events associated with an IP address of the device 160interacting with the retailer's site 150 and an associated start time ofthe session.

At this point then, the enhanced or adjusted spot view data 176 may bematched with the enhanced or adjusted user interaction event data 192 toassociate (if possible) a user's viewing of a spot with that user'sinteractions with the retailer's website 150 (STEP 194). Generally, thismatching may be accomplished by matching the IP address associated withspot viewing data in the enhanced or adjusted viewing data 176 with thecorresponding IP address of the enhanced or adjusted user interactionevent data 192 to associate a user's viewing of a spot with acorresponding session with an associated retailer's site 150.

Accordingly, this may be a first merge step that is an exploded crossjoin between two data sets based on IP address matches. Any matcheswhere the determined spot viewing time for a spot view in the enhancedor adjusted spot view data 176 occurs after the session start time asdetermined for a matching session in the enhanced or adjusted userinteraction event data 192 may be discarded.

Next, in cases where there are still multiple matches between one ormore spot views in the enhanced or adjusted spot view data 176 and oneor more matching session in the enhanced or adjusted user interactionevent data 192, the match may be kept between the matching session orevent whose start time is closet in time to the spot viewing time of thematching spot viewing event of enhanced or adjusted spot view data 176.Thus, the matching spot view data and user interaction event data 196may be stored at the quantification system 170.

This matching (or merged) data 196 and other data obtained or determinedby the quantification system 170 may be utilized for a variety ofpurposes. For example, the matching data 196 can be used to establish abaseline of website visitor traffic. There can be many ways to establishsuch a baseline. For example, a website visitor traffic baseline can beestablished by taking a moving average of UVs to a website throughoutthe day or the week and excluding those that clearly do not come becauseof any TV spots (e.g., UVs that came to the website through a link in anemail, through a referring website, etc.). This is referred to as a UVbaseline and may be particular for TV conversion attribution computation(because it excludes non-TV influences). The UV baseline essentiallyplots, over the course of a timespan, what is considered normal websitevisitor traffic to a website relative to TV spots, as reflected in thenumber of UVs to the website.

Different TV spots may contain different messages to TV viewers. Forinstance, some TV spots may be directed to a physical product, whilesome TV spots may be directed to a new app install. For the purpose ofillustration, suppose a TV spot calls for viewers taking a certainaction through a particular website (e.g., visit a website, buy aproduct through the website, download and install an app from thewebsite, etc.), a “conversion” event occurs when a UV to the particularwebsite takes that action (e.g., the UV visited the website, the UVpurchased the product through the website, the UV downloaded andinstalled the app through the website, etc.). Such conversions can beconsidered as a conversion lift.

“Lift” is a quality metric for measuring a spot in the context of aparticular type of campaign. Since the merged data contains the timeperiod by time period (e.g., minute-by-minute) cohort (i.e., all the UVsto the website during the same minute), the quantification system cancalculate a conversion rate based on the cohort and determine thevisitor lift relative to the UV baseline. In some embodiments, this canbe done by examining every minute in a day for a number of days duringwhich offline data and online data have been aggregated and merged withregard to a particular website and computing a conversion rate for eachminute. Thus the conversion rate during a time window for a particularTV spot may be isolated.

Because the cohort based on which the conversion rate is calculate mayinclude both TV viewers and non-TV viewers, the initial result from thiscalculation can be skewed. For example, suppose a spot aired on a TVnetwork at 6:20 PM and a lift (an increase in the conversion rate)occurred shortly after 6:20 PM, it is possible that the lift can beattributed to the spot that aired at 6:20 PM. However, it is unclear howmuch of that lift can actually be attributed to the spot that aired at6:20 PM.

One way to eliminate this skew and quantify the attribution of such alift to TV conversion is to examine session timestamps, correlatesession timestamps to spot airing data, and assign a timestamp to aconversion event that occurred on a website. This can be done byexamining every minute of every UV's visit to the website within awindow of time when a spot aired on a TV network. Following the aboveexample in which the spot aired on a TV network at 6:20 PM, thequantification system may operate to examine UV sessions initiated atthe website within a time window (e.g., five minute) starting at 6:20PM; thus 6:20 PM, 6:21 PM, 6:23 PM, 6:24 PM, and 6:25 PM.

The quantification system may track each UV session and determinewhether a conversion occurs during a UV session. If a conversion occursduring a UV session (a conversion event), the quantification system mayassign the start of the UV session as a timestamp associated with theconversion event. For example, suppose the spot aired on a TV network at6:20 PM on Day One with a message for viewers to donate to a charity. AUV initiated a session with the charity's website at 6:25 PM on Day Oneand subsequently made a donation (e.g., a hour, a day, or even a weeklater).

In this example, the quantification system may assign 6:25 PM on Day Oneas the timestamp for the conversion event. In some embodiments, if thereis a time gap between visits by the same UV to the website, thequantification system may use the latest (active) session timestamp asthe timestamp for a conversion event. That is, if there are multiplesessions, the quantification system may select the most recent one ifthe most recent one and the one before it has a temporal gap that islarger than a predetermined threshold (e.g., three months, six months,one year, etc.). This conversion event timestamp assignment isirrespective of the spot airing date/time.

Now that each conversion event has an assigned timestamp, thequantification system 170 may operate to determine conversion rates ofUVs to a website during a time window defined relative to a spot airingtime. The quantification system may keep data for every UV to aparticular website (from the online data), documenting when UVs visitthe website, when their sessions started, and when or if a conversionevent took place. The quantification system can compare, on a minute byminute basis, user interactions with the website relative to when asport aired (e.g., at 6:25 PM). Using a fixed window (e.g., five minutesfor this website), anything occurred during 6:22-6:27 PM, thequantification system can determine how UVs visited the website in thatwindow of time and how many conversion events have taken place in thatwindow of time. This gives an overall conversion rate in the TV window(which includes both TV responders and non-TV responders). Suppose thetotal of UVs is 100 UVs to the website over the five-minute window, and10 of them made a purchase. This results in a 10% conversion rate ingeneral for that window. This can be defined as follows:Conversion rate=number of conversions that occurred as a result of UVsthat started their session within a time window/number of UVs within thetime window.

The quantification system 170 then determines how much of thisconversion rate can be attributed to TV (i.e., the conversion rate of TVresponders). In some embodiments, this can be done by isolating the liftfrom TV on a minute by minute basis, applying a new alpha factor (a),and adjusting the conversion rate using the alpha factor. In thisdisclosure, the alpha factor represents a ratio between the conversionrate of TV UVs and the conversion rate of non-TV UVs.

As discussed above, a lift has been computed and associated with eachminute the quantification system has data for a website (e.g., everyminute across all the days and months running a campaign). Also assignedto each minute is a percentage of lift that came from immediate TVresponders, from 0 to 100%. Below is an example of how this percentagecan be determined.

Suppose the quantification system 170 has computed the total conversionrate (e.g., the number of conversion events divided by the number of UVsper minute). When the percentage of TV users is 0, the conversion rateis 2%. When the percentage of TV users is 100, the conversion rate is1%. Therefore, in this example, the TV users have half the conversionrate in general than the non-TV users. This ratio is represented by thealpha factor. The alpha factor can be determined using an ordinary leastsquares (OLS) regression technique as follows:y=α ₀ +αX+ε,where y=vector of conversion rates by minute and X=vector of % TV users(lift/UV) per minute.

That is, the quantification system 170 examines the percentage of TVusers for each of those minutes and solves what that alpha factor is at100% TV lift to 0 TV lift.

Once the alpha factor is determined, the conversion rate for TVresponders can be determined by multiplying the number of conversions bya factor lambda (λ). This is reflected in the new equation below:

${{{conv}{{rate}({tv})}} = {{r\left( {tv} \right)} = {\sum\limits_{t}\frac{\lambda_{t}c_{t}}{m_{t}}}}},$where

${\lambda = \frac{\alpha p}{{\alpha p} + \left( {1 - p} \right)}},$m=lift, and

-   -   p=likelihood of user being from

${tv} = \frac{lift}{UV}$for that minute,

-   -   α=ratio of tv conversion rate to nontv conversion rate,

By computing this equation, the quantification system 170 can determinethe conversion that can be attributed to TV responders. That is, thequantification system 170 is operable to examine the total conversionrate relative to a specific lift, multiplying those conversions by thelambda factor, which is the alpha factor times p divided by the alphafactor times p plus one minus p. Put another way, the conversionstracked and ascribed to those users in a TV window cohort are adjustedby the proportion of TV viewer conversions responding in that window;(e.g., proportion of conversions associated with TV viewer lift dividedby the proportion of conversions associated with TV viewer lift plus theproportion of conversions associated with non-TV viewers. This is theconversion rate of immediate TV responders.

The quantification system 170 may also use the matching data 196 for thedetermination of a latency factor (STEP 197). The determined latencyfactor can then be applied to a variety of metrics (e.g., the lift orconversion rate as discussed above) to adjust those metrics or determinenew metrics of data (STEP 198). The metrics or data can then bepresented in a user interface such as a dashboard for presentation oncomputing devices coupled to the quantification system 170 (STEP 199).

In particular, as has been discussed, data associated with event datafrom the retailer's online presence and the IP address of the user'sinteractions with the offline network (e.g., as determined from theuser's “smart” device used to access the offline network) can beobtained or determined such that the user's online interactions(including interaction with a retailer's presence in the online network)can be correlated or otherwise associated with the user's interactionwith the offline network.

This data can then be utilized for a variety of purposes and ends byquantification systems. As but one example, this data can be used by thequantification system to calculate one or more metrics associated withboth a creative, or group of creatives, aired by the retailer on theoffline network and the retailer's presence on the online network. Thesemetrics may include efficiency, relative performance, lift or response,conversion rate, spend per conversion, or other metrics as discussed.These metrics, or data determined therefrom, can be displayed to a useraccessing the quantification system in an intuitive interface to allowthe user to assess the efficacy or other insights into the retailer'screatives.

There are, however, a number of difficulties that occur in determiningsuch metrics. One such difficulty is the difficulty of measuring thelatent effects (e.g., the effect after some time period) of the airingof a creative on one or more metrics, including metrics associated witha user's interactions with the retailer's online presence. This isbecause the retailer's online presence is almost always available to auser, thus a user may access the retailer's online presence directlyafter seeing a creative, or may wait some undetermined amount of timebefore accessing the retailer's online presence. Also, in many cases,offline content is accessed at different times than the original airingof the creative. For example, a user may record a TV program and watchthat program (and any associated creatives) subsequently to the originaltime the program aired. That user may then access the retailer's onlinepresence at some undetermined amount of time after the user watched thecreative (which is also some undetermined amount of time after thecreative actually aired).

It is thus desired to quantify metrics using a latency factor which mayaccount for the effects of the airing of a creative that occur outsideof some baseline window (e.g., time period, such as 5 minutes, 10minutes, 30 minutes, etc.) following the original airing of thecreative. Such a latency factor may be associated with, or otherwisereflect, an expected response to an airing of a spot outside of thebassline window (e.g., relative to a response within the baselinewindow) vis a vis the retailer's online presence. Thus, a latency factorcan be applied to almost any other determined metric to estimate oradjust that metric to account for user's interactions that occur outsideof the baseline window and within a selected time period (e.g., 30 days,60 days, a quarter, a year, etc.).

By determining such a latency factor, users can access and determine thepotential response from a creative more quickly, improving the computerperformance by allowing metrics and other data to be determined morequickly and interfaces to be generated more quickly. It also can providethe capability of measuring delayed results on a granular level. Otherapproaches either will not be able to gauge the full response to anindividual spot, or will only be able to estimate the full response onan aggregated level. Such a latency factor may be usefully applied to anumber of metrics that may be determined by a quantification system,including for example traffic lift or conversion.

Turning first to an embodiment of determining a latency factor fortraffic lift, some context may be useful. As explained above, and inmore detail in U.S. patent application Ser. No. 16/365,448, filed Mar.26, 2019, entitled “SYSTEMS AND METHODS FOR ATTRIBUTING TV CONVERSIONS”by Swinson et al. and U.S. patent application Ser. No. 16/360,456, filedMar. 21, 2019, entitled “SYSTEMS AND METHODS FOR DEBIASING MEDIACREATIVE EFFICIENCY,” by Swinson et al., the entire contents are fullyincorporated herein for all purposes, for the TV advertising industry,one embodiment may measure the metric of efficiency using traffic lift.Assuming t is the time (minute), and non-TV normal traffic is F(t). Oncean ad is aired at t0, new traffic changes from F(t) into G(t), and acreative's contribution is the difference between G(t) and F(t).

One challenge in this calculation process is that only the real traffic(e.g., to a retailer's online presence such as a website) may beobserved. For example, assume the creative (also referred to as an ad,spot or advertisement) is aired at t0. Before t0, the traffic observedis F(t) (t<t0); after t0, the traffic observed is G(t) (t>=t0), however,no F(t) (t>=t0) can be observed. Due to the fact that the traffic isusually a stochastic process with random noise, and also many TV spotsmay have aired at similar times, estimating F(t) (t>=t0) directly isusually an incredibly difficult, if not impossible, task.

Meanwhile, it is observed that there is usually a strong traffic spikefollowing a creative airing (usually within 5 or 10 minutes), whichcreates an opportunity to use immediate lift in traffic to theretailer's online presence to estimate or determine efficiency for thecreative. This becomes a standard in embodiments to measure creativeefficiency. The “immediate lift” approach to embodiments leverages asmoothing algorithm to create a baseline, which mimics the F(t) beforeand after the airing, and the spike area above the baseline will beconsidered the contribution to the traffic from the creative. Oneembodiment of an interface graphically depicting a relationship betweenthe spike in traffic on a retailer's site and the airing of creatines onvarious TV networks (e.g., VEL, LMN, SYFY) is depicted in FIGS. 2A and2B.

Such an approach is useful in embodiments, with a caveat: the long term(or latent) TV advertising effect is ignored with this method, becauseit factors into the baseline. Since TV creatives usually create animpression over a product that can drive people to visit a website orinstall an app long after they viewed the ads, such long term effectshould not be ignored and needs to be considered when measuring TVefficiency.

Referring to FIG. 3 , as an example, embodiments of quantificationsystems may track viewers of a retailer's ad via IP addresses, asdiscussed. In the following example, there are 12*9=108 IPs reachablevia TV, 12*2=24 IPs have viewed [retailer A] ad. Within the following 5minutes, there are 2 IPs with [client A] app download (e.g., access toretailer's online presence); all of those are within IPs who viewed[client A] ad. Now, the immediate lift is 2 installations (e.g., accessto the online presence) for this ad, comparing with no ad.

However, after 10 days, naturally, IPs without ad impressions may alsoinstall the app (e.g., here assume 4 out of 84), while those with adimpressions have 5 out of 24 installed the app. The real value or metricmeasuring the efficiency or lift created by the TV ad is thus(5/24−4/84)*24=3.86. In this example, the latency factor may be3.86/2=1.93 for [client A] ad efficiency.

More generally, then, embodiments of this methodology are used byembodiments of a quantification system to estimate the latency factor ofa retailer's creatives. In particular, certain embodiments ofquantification systems may leverage rich data sets (TV viewing andretailer's online access data (e.g., app install data) and statistics(e.g., conditional probability distribution with or withoutbootstrapping sampling (as will be discussed)), to determine a latencyfactor to help retailers and associated creators of creatines betterunderstand the efficiency or efficacy of their advertising.

Looking at FIG. 4 then, a flow diagram for one embodiment of a methodfor determining a latency factor that may be employed by aquantification system is depicted. As discussed above, thequantification system may obtain data from a variety of sources andmerge or match such data (STEPS 410, 420). Certain embodiments mayutilize three types of data to determine the latency factor of TVadvertising for any given product, application, retailer, etc. 1. TV adspot airing information (date/time, station, content), 2. TV viewershipdata with IP addresses or device identifiers (ids), and 3. Clickstreamdata: app-install or web-browsing activities with associated IPaddresses or device ids. Note here that for simplicity purpose inexplaining certain embodiments, application installs may be the metricor factor utilized as an example with IP addresses as the matchkeybetween data from different sources. It will be noted that web-browsingor user view or other latency factors can be determined in a similarmanner. Other data may also be used to correlate or associate data fromdifferent sources, including device ids for user's other devices orother data.

In any event, from enhanced data at the quantification system, thefollowing information can be determined by matching or merging onlineand offline data: whether an IP address was exposed to the specificadvertisement and whether this IP has installed the application.Ideally, each IP address will be associated with only one device (withapplication download activity); however, in reality, there are cases inwhich multiple devices are associated with one IP. In one embodiment,the data sets collected or determined have the following structure, foreach specific client:

IP address Client last_view_time app_install_time multiplicity 0.0.0.0 A2017 Jul. 1 N/A N/A 00:00:00 0.0.0.1 B 2017 Jul. 1 2017 Jul. 1 101:00:00 01:03:00 0.0.0.1 A 2017 Jul. 1 2017 Jul. 1 4 02:00:00 02:10:00

Here, if one IP address has no app installation, then the last_view_timeis the last time an ad was aired and viewed by that IP address; if oneIP address has one app installation, then the last_view_time is theprevious time an ad was aired and viewed by that IP address; if one IPaddress has multiple app installations at different times, then thelast_view_time is the previous time for each app installation when an adwas aired and viewed by that IP address. To avoid cases where a publicIP is shared by multiple devices, a strict rule may be applied by thequantification system to remove those IPs with more than one appinstallation. This rule can be changed in other embodiments to adifferent threshold of app installation or device id combination per IPaddress. Effectively, the data becomes following:

IP address Client last_view_time app_install_time 0.0.0.0 A 2017 Jul. 100:00:00 N/A 0.0.0.1 B 2017 Jul. 1 01:00:00 2017 Jul. 1 01:03:00

When the latency factor is evaluated for a retailer, for example[retailer A], IPs that viewed that retailer's [retailer A's] ad (or aparticular one of that retailer's ads) are called Targeted IPs. OtherIPs (e.g., IPs that did not view the retailer's ad or the particularretailer's ad), or a subset of other IPs, are called Reference IPs. Aset of Target IPs and Reference IPs can thus be determined based onwhether an IP has viewed the retailer's ad or a particular of theretailer's ad (STEP 430).

Embodiments may thus determine a conditional probability. Whileembodiments and examples have been discussed with respect to theinstallation of an application it will be noted that substantiallysimilar embodiments may be equally effectively applied to views oraccess to retailer's online presence such as website views orconversions or purchases of items through such an online presence.

Since the determination of a latency factor will always be conducted ona specific date or time, it may not be possible to determine whether anIP will have any app-installs in the future (e.g., subsequently to thedetermination). So the max-time is defined as from the time an IP wasexposed to this advertisement (t0), until now (tnow) when the mostrecent app installation data is collected):t max=tnow−t0

The individual (uniquely mapped to an IP address) can choose to install(e.g., the app or view the website) or not. If he still has notinstalled the app or viewed the website at t_(now), the effective viewtime is t_(now)-t₀. If the individual has installed the app or viewedthe website at t_(install) (before t_(now)), then the effective viewtime is:tinstall−t0. deff=min(tnow,tinstall)−t0

This is the effective duration after an ad has aired that a user hasinstalled the app. The conditional probability for installation within aspecific time range can be expressed as:P(install|t0<=t<=t1)=Ninstalled/Nviewed

How many installations happen by t1 (before t_(now)), after the ad wasaired on t0 can then be evaluated. This is determined by:

${P\left( {{install}{❘{t_{0}<=t<=t_{1}}}} \right)} = {\sum\frac{N\left( {{install}{❘{t_{0}<=t<=t_{1}}}} \right)}{N\left( {{viewed}{❘{t_{0}<=t<=t_{1}}}} \right)}}$

With the cumulative conditional probability, the percentage of TVviewers to install this app can then be determined, up to a specifictime. Such a determination will be applied over Targeted IPs. Then, toremove organic installation, the same determination over Reference IPscan be determined and their cumulative conditional probabilitydetermined:P(install|t0<=t<=t1 & noview)where this population of users viewed another ad over the same timerange, but did not view the retailer's ad.

The difference will be the incremental installation caused by thisairing of the creative:Preal(install|t0<=t<=t1)=P(install|t0<=t<=t1 & view)−P(install|t0<=t<=t1& noview)

This determination is expressed graphically in FIG. 5 . Here, there aretwo different data sets. The Target IPs for a creative (or a set ofcreatives, type of creative, segment of a creative, creatives on aparticular channel, etc.), are the set of IPs that responded (e.g.,interacted with the retailer's online presence in a particular manner).From this set of Target IPs, some number responded during the baselinewindow (e.g., some amount of time after the airing of the creative, suchas 5 minutes) while some number responded outside of the baseline windowbut within the time horizon or overall time window (e.g., 30 days). Thedata from these Target IPs can be used to create the cumulativeprobability curve for the Target IPs (STEP 440), where the X axis is thetime since the creative aired and the Y axis is a percentage or ratio ofthose viewers who saw the ad that responded.

It will be noted that this Target IP curve may, in some embodiments,overestimate the response because some of those users would respond(e.g., access the retailer's online presence) regardless of theirviewing of the creative. To control for this population, a set ofReference IPs may be used. These Reference IPs may be for users who sawcreatines on similar or other networks at similar times and interactedwith the retailer's online presence (e.g., responded). This will give anapproximation of the users who would have responded in any event (e.g.,without viewing the creative of interest).

Thus, from this set of Reference IPs, some number responded during thebaseline window (e.g., some amount of time after the airing of thecreative, such as 5 minutes), while some number responded outside of thebaseline window but within the time horizon or overall time window(e.g., 10 days, 30 days, etc.). The data from these Reference IPs can beused to create the cumulative probability curve for the Reference IPs(STEP 450), where the X axis is the time since the creative aired andthe Y axis is a percentage or ratio of those viewers that saw thecreative that responded.

By subtracting the Reference IP curve from the Target IP curve, acumulative probability difference curve can be determined (e.g., whichaccounts for those users who would have responded regardless of viewingof the creative) (STEP 460). The latency factor can then be determinedfrom this cumulative probability difference curve (STEP 470) by dividingthe response percentage, or number, at the expiration of the time of thebaseline window (e.g., 5 minutes in the example) as determined from thecumulative probability difference curve by the response percentage ornumber at the end of the time horizon or overall window (e.g., in thisexample 10 days) as determined from cumulative probability differencecurve. An example interface presenting an actual curve is depicted inFIGS. 5 and 6 along with an interface presenting the latency or drag(used interchangeably) for a creative in FIG. 7 .

Once a latency factor is determined, this latency factor can then beapplied to a metric such as immediate response to estimate or determinea value for a metric (e.g., unique visitors, application installs, etc.)over the time horizon of overall window. For example, if a latencyfactor is as computed above, an example might be:

${{latency}{factor}} = {\frac{{total}{response}}{{imme}{diate}{response}} = {\frac{0.295\%}{0.133\%} = {{2.2}2}}}$

This latency factor may inform a user such as a retailer how muchadditional delayed response they can expect (e.g., to their creative orat their online presence). In the above example, that delayed responseis 1.22 times the (immediate response). So, basically:delayed response=(latency factor−1)*immediate_response

So, in the case of measuring a lift in unique visitors to a website, itwould be:UV_lift_delayed=(latency factor−1)*UV_lift_immediate

Embodiments may assume that the metric (e.g., the app installation orwebsite views) for both Target and Reference IPs have similarinstallation patterns across the day, so that the cumulativeinstallation over the whole period time is comparable. However, it ispossible that two groups have different installation patterns (differenthours, different day of week, etc.), which will impact the twocumulative probability curves. This situation could be, for example, dueto differences in the timing of creative viewership between the twosamples (e.g., consumers viewing ads in the evening may be more likelyto install an app or view a website).

To account for this possibility, in certain embodiments sampling via thebootstrap technique may be applied to the reference viewershippopulation (e.g., Reference IPs) based on the retailer's viewershippopulation, so that the reference retailer population per hour per dayof week closely represents the same distribution as the targetedretailer population. In this way, no population bias will be introducedby individuals that viewed the retailer's ad compared with individualsthat viewed ads other than the retailer's, and the comparison betweenthe two populations (e.g., Target IPs and Reference IPs) is meaningful.

In other words, in some embodiments, the quantification system maycreate a sample set of Reference IPs (e.g., DF_REF) approximately thesame size, or the same size, as the set of Target IPs (e.g., DF) bymatching the number of viewers by time period (e.g., an hour) of thenon-retailer spots with a replacement from the DF_REF sample. Suchbootstrapping approach may be accomplished by first generating alast-view date-time period (e.g., hour) histogram for both Target andReference IPs. Following the generation of the histogram and given afallback logic, the Reference IP creation will resample with replacementover each time period (e.g., hour) bucket to achieve the desireddistribution. This fallback logic may include filtering on a cluster ofsimilar network based on the following ordered set of rules:

Fallback Logic Rules 1 same hour 2 same hour, same day of week 3 +/−1hour, same day 4 +/−1 hour, same day of week

Thus, in embodiments: given two datasets DF of Target IPs and DF_REF ofReference IPs, determine the sample DF according to the distribution ofthe DF_REF to get DF_NEW. Then, calculate a conditional probability forboth DF_REF and DF_NEW. The cumulative probability difference betweenDF_REF and DF_NEW will then become the latency curve and the latencyfactor for a time period (e.g., 10 days) equals the ratio of percentagesbetween the point where t equals the time period on the latency curve(e.g., t=10 day) and t equals the outer limit of the baseline timeperiod on the latency curve (e.g., t=5 min) (e.g., such as depicted inFIG. 5 ).

In another embodiment, a predicted number or curve of users who wouldhave responded (e.g., access the online presence) regardless of theirviewing of a spot may be quantified using a simulation based on datacollected from the quantification system. FIG. 10 is a flow diagram forone embodiment of such a method for simulation. As discussed above, datacollected from the quantification system may include content data suchas who (e.g., device id or IP address) is watching which program onwhich network at what time interval and spot view data: who (e.g.,device id or IP address) saw which retailer's spot during which programon which network starting at which time and ending at which time.

A baseline of responders (e.g., a baseline or Reference IP responsecurve) may be constructed using a simulation. In this case, as thebaseline is being simulated (e.g., users that would respond regardlessof their viewing of the creative), their response time (e.g. time ofvisiting the retailer's online presence) should be independent ofwhether or when they viewed the creative).

Accordingly, in one embodiment, the time range of interest (e.g., fromthe start_time to end_time) can be determined (STEP 1010). This may befrom the spot airing time or baseline time window time (start_time) tosome overall time window (end_time). Some number of (e.g., N) users(e.g., device id or IP address) may be randomly selected during thistime range (STEP 1020). All the spot view data during this time rangecan be determined (STEP 1030).

Then for each one of the users (N) above, it can be determined if theuser has seen a spot (e.g., any spot) for the retailer or not (STEP1040). If the user has not viewed a spot for the retailer, that user maybe excluded or dropped from the set of users (STEP 1042) and the nextuser evaluated if there are any remaining users in the set. Otherwise,if the user has viewed a spot for the retailer, a time between thestart_time and end_time weighted by the traffic trend pattern may beselected as the response time for the user (STEP 1044). It can then bedetermined if there is any spot that starts before the determinedresponse time (STEP 1050).

If there is no spot that starts before the determined response time thatuser may be excluded or dropped from the set of users (STEP 1042) andthe next user evaluated if there are any remaining users in the set.Otherwise, if there is a single spot that starts before the determinedresponse time use the difference between the spot start time andresponse time as the response latency (STEP 1052), while if there aremultiple spots that start before the determined response time use thesmallest difference between the spot start times and response time asthe response latency for that user (STEP 1054). The random sampling andrefinement may be repeated to determine users for each run (e.g., eachrepeat of LOOP 1056).

All the remaining users from the one or more runs (e.g., repetition ofLOOP 1056) may be collected and grouped by response latency (accordingto some time period such as hours or days) and count the users in eachgroup (called simulated response latency profile) (STEP 1060). The timeperiod utilized may be determined based on the amount of data availableand a desired signal-to-noise ration. The simulated response latencyprofile can then be scaled so that it matches the observed dailyresponse latency profile on the tail of the curve (STEP 1070). Thisscaled simulated response latency profile may then be used as thebaseline curve (e.g., for the Reference IP set).

Now with reference to FIG. 11 , a visual depiction of an example ofobserved latency profiles and scaled simulated response latency profilesare determined. In the example curve 1110 is the observed (here daily)response latency profile while curve 1120 is the scaled simulated (heredaily) response lag profile. In this example, the tail is defined aslatencyday>=30 days. However, it will be noted here that in general, thecutoff latencyday (e.g., here 30 days) as well as the cutoff time period(e.g., minute) for immediate responses (e.g., here 5 minutes) may varyon case by case, and may be dependent on the context, including theretailer or spot of interest.

The simulated (daily) response latency profile 1120 in this example isthus scaled to the magnitude shown so that it matches the observed curve1110 after latencyday>=30 days. In the depicted example the dotted curve1120 between 0 and 30 days is the baseline. The differences between thecurves 1110, 1120 between 0 and 30 days are the daily lift attributableto TV spots. Thus, in this example, Latency factor=Overall Lift (0 to 30days)/Immediate Lift (0 to 5 minutes) where Overall Lift=SUM(curve1110−curve 1120) from 0 to 30 days and Immediate Lift=SUM(curve 1110 atminute level−curve 1120 at minute level) from 0 to 5 minutes.

Certain modifications or alternatives may be utilized in someembodiments, for example DVR data may be utilized to adjust the latencyfactor, as it can be determined when creatines were seen and when a userresponded based on actual viewing. TV viewership data can be used tocompute visitor latency in a number of ways.

Additionally, in some embodiments, latency factors can be determined forcertain segments of data. Specifically, in certain embodiments, theselatency factors can be separated out to produce for different segmentsof the population by filtering the data first by that segment andcomputing as above. For example, to gauge the latency factor for oneparticular channel (e.g., network), the quantification system can firstfilter the data to only those viewers of that network. Then, thecalculations can be performed as before.

Moreover, in certain embodiments a latency factor can be determined for,and applied to, the metric of conversions or sales lift. Thisconversions latency may be performed by, for the two populations (againDF comprising Target IPs that viewed a particular creative and DF_REFcomprising Reference IPs that did not view that creative), attaching ordetermining a field for their respective sales per user after viewingthe creative. Then the quantification system can use the session datetimestamp as before. Thus, when determining a latency factor forconversion the only difference from the determination of latency fromthat described above is comparing sales lift instead of installs or UVlift.

In particular, here, in one embodiment the conversion rate on theadditional sales can be determined as:(sales_DF−sales_DF_REF)/UV_lift

While the conversion rate from the above can be used directly, this datacan also be used to compute one of the factors that can be used in thealternate (e.g., more real-time measure of computation). This can bedone by computing the conversion rate of DF_REF by:sales_DF_REF/UV_lift

As conversion rates for both populations have been obtained, theconversion rates of the population (DF or Target IPs) that viewed thecreative versus the population (DF_REF or Reference IPs) that did notview the creative can be determined. This gives an alternative view ofthe factor alpha from the determined conversions.

Thus, similar to embodiments focused on quantifying delayed response,embodiments can also quantify conversions based on delayed responders toa retailer's creative. FIG. 9 is a flow diagram depicting one embodimentof a method for using a latency factor to quantify a metric. Initially,at step responders may be determined or isolated in the matched ormerged data set (STEP 910). This determination may be done asbefore—identifying those who responded to a particular spot via matchingIP addresses between the target viewership dataset and the clientclickstream dataset. This set may be referred to as TGT resp.

Next, it can be determined which of these responders responded withinthe TV attribution or baseline window (e.g., first 5, 10, 20 minutesetc. after a creative airs) (STEP 920). Embodiments may analyze bothsets, but embodiments may distinguish between TGT_resp_immediate (e.g.,those IPs of TGT_resp that responded within the baseline window) andTGT_resp delayed (e.g., those IPs of TGT_resp that responded outside thebaseline window).

The subpopulation of these target responders that ‘converted’ (e.g.,made a purchase of one or more designated products or services) can thenbe determined for each of the two above populations (STEP 930). Thesesub populations may be referred to as TGT_resp_immediate_conv (e.g.,those IPs of TGT_resp_immediate that converted) and TGT_resp delayedcony (e.g., those IPs of TGT_resp delayed that converted). In oneembodiment, the timing of the conversion event may be determined foreach of these determined conversions. For example, in one embodiment arule may be applied such that the conversion event must have occurredafter the time of viewing the spot for it to be credited in thisdetermination. An overall time window (e.g., 30 days) may also be usedover which to count a conversion (e.g., no conversion occurring laterthan this overall time window will be utilized). This capping ofconversion performance may help prevent censorship bias (some datapoints having more time to conversion than others).

It should be noted here that in many embodiments the designatedconversion metric is a purchase, but the conversion metric could bealmost any desired metric, including any deeper “funnel” event on awebsite, such as membership registration, or some other specified event.The identifications of these events may be straightforward, as thedeeper funnel conversion event may be tracked in the retailer'sanalytics. For example, the quantification system may track such eventsthrough a pixel placed on the retailer's site or pages thereof, or theuse of other pixeling or tracking services (e.g., Google Analytics,Adobe Omniture, etc.).

Responders in the reference dataset (e.g., Reference IPs) can also bedetermined (STEP 940). This is done as discussed previously—identifyingthose who responded via matching IP addresses between the referenceviewership dataset and the client clickstream dataset. This set may bereferred to as REF resp.

It can then be determined which of these responders of the Reference IPSresponded within the TV attribution or baseline window (e.g., first 5,10, 20 minutes etc. after a creative airs) (STEP 950). Embodiments mayanalyze both sets, but embodiments may distinguish betweenREF_resp_immediate (e.g., those IPs of REF_resp that responded withinthe baseline window) and REF_resp_delayed (e.g., those IPs of REF_respthat responded outside the baseline window).

The subpopulation of these reference responders that ‘converted’ (e.g.,made a purchase of one or more designated products or services) can thenbe determined for each of the two above populations (STEP 960). Thesesub populations may be referred to as REF_resp immediate cony (e.g.,those IPs of REF_resp immediate that converted) and REF_resp_delayedcony (e.g., those IPs of REF_resp_delayed that converted). Again, in oneembodiment, the timing of the conversion event may be determined foreach of these determined conversions. For example, in one embodiment arule may be applied such that the conversion event must have occurredafter the time of viewing the spot for it to be credited in thisdetermination. An overall time window (e.g., 30 days) may also be usedover which to count a conversion (e.g., no conversion occurring laterthan this overall time window will be utilized).

The segmented conversion rates can then be determined for the determinedpopulations (STEP 970). Here, the immediate response Target Populationconversion rate can be determined by:

 TGT_immed_convR = TGT_resp_immediate_conv ÷ TGT_resp_immediate  Immediate response Reference Population conversion rate: REF_immed_convR = REF_resp_immediate_conv ÷ REF_resp_immediate   Delayed responder Target Population conversion rate:   TGT_delay_convR = TGT_resp_delay_conv ÷ TGT_resp_delay    Delayedresponder Reference Population conversion rate:    REF_delay_convR =REF_resp_delay_conv ÷ REF_resp_delay   Incremental immediate responseconversion rate:    Incr_immed_convR     = (TGT_resp_immediate_conv    − REF_resp_immediate_conv)/(TGT_resp_immediate     −REF_resp_immediate)   Incremental delayed response conversion rate:Incr_delayed_convR    = (TGT_resp_delayed_conv    −REF_resp_delayed_conv)/(TGT_resp_delayed − REF_resp_delayed)

The determined conversion rates can then be validated versus the overallpopulation (STEP 980). In particular, since the population of the dataset of a quantification system may be a somewhat different populationthan the overall TV viewer population, the system can now determine howsimilar the conversion rates are between Incr_immed_convR and theincremental conversion rate that was determined on the whole population(e.g., based on the determination of conversion as discussed above). Thedifferences here are usually due to three factors: noise, populationdifferences, and differences in assumptions about active users.

The delayed versus immediate conversion rate ratio can then bedetermined (STEP 99). The ratio, convR_ratio=Incr_delayconvR/Incr_immed_convR can be computed and applied to the conversionrate of the entire population (e.g., the determination of which isdiscussed above).

In one embodiment, the data may utilize a low-volume sample or ahigh-volume sample. As may be ascertained by review of the determinationof conversion rate as set forth herein, the determination of conversionattempts to derive the incremental conversion rate that the systemcalculates on the whole population. However, as described above, thereare some statistical assumptions in that computation. With this ‘closedloop’ approach (where the ‘closed loop’ sample is defined as the samplefor which viewership data is available), the determination for theretailer may be limited to a very small population based on thepercentage of viewership data that is available. Thus, the result can beless reliable if that population is too small.

Accordingly, if there is a large viewership population, then the systemmay rely on the incr_immed_convR as described above (e.g., incrementalimmediate response conversion rate), previously. However, aquantification system may make a determination that the computation ofconversion rate as detailed may be more accurate (e.g., if theviewership data sample is very small (e.g., 1% of the total), then thesystem could alternatively leverage the incr_immed_convR as computed forthe conversion rate as detailed above. If this is the case, then thesystem may derive the incr_delay_convR. That derivation would becomputed as

Incr_delay_convR = (convR_ratio * incr_immed_convR) where convR_ratio iscomputed as convR_ratio = Incr_delay_convR/Incr_immed_convR

As may be imagined, the determination of latency factors for variousmetrics, such as user views or conversion, may be usefully applied in avariety of contexts for a variety of purposes, including for example,lending retailer's insights into the efficacy and cost structure oftheir creatives and campaigns associated with those creatives.

So, for example, some use cases for the latency factor may include useof the response latency factor to determine cost per view (CPV):

CPV = cost/UV_lift = cost/(UV_immed_lift * drag_factor)

Again, the objective for most retailers may be ultimately understandingthe cost per sale (CPS) for each of their creatives, networks, andcampaign as a whole. The calculation of CPS can be determined asCPS=CPV/convRate. So, using the information (as computed above), thesystem can compute CPS as such.

Based on the above, the system can derive the total conversion rate:

convRate = (incr_immed_convR * UV_immed_lift + incr_delay_convR *UV_delay_lift)/(UV_immed_lift + UV_delay_lift)

This can be rewritten mathematically as:

convRate = (incr_immed_convR * UV_immed_lift * (1 + convR_ratio * (drag_factor − 1))/(UV_immed_lift * drag_factor_ − CPS =  CPV/convRate

Note that this is essentially the same as:

CPS = cost/sale = cost/(sales_from_immed_responders +sales_from_delyaed_responders)but derived from the various factors as discussed above.

The invention and the various features and advantageous details thereofare explained more fully with reference to the nonlimiting embodimentsthat are illustrated in the accompanying drawings and detailed in thefollowing description. Descriptions of well known starting materials,processing techniques, components and equipment are omitted so as not tounnecessarily obscure the invention in detail. It should be understood,however, that the detailed description and the specific examples, whileindicating preferred embodiments of the invention, are given by way ofillustration only and not by way of limitation. Various substitutions,modifications, additions and/or rearrangements within the spirit and/orscope of the underlying inventive concept will become apparent to thoseskilled in the art from this disclosure. Embodiments discussed hereincan be implemented in suitable computer-executable instructions that mayreside on a computer readable medium (e.g., a hard drive (HD)), hardwarecircuitry or the like, or any combination.

Embodiments of a hardware architecture for implementing certainembodiments is described herein. One embodiment can include one or morecomputers communicatively coupled to a network.

At least portions of the functionalities or processes described hereincan be implemented in suitable computer-executable instructions. Thecomputer-executable instructions may be stored as software codecomponents or modules on one or more computer readable media (such asnon-volatile memories, volatile memories, direct access storage drive(DASD) arrays, magnetic tapes, floppy diskettes, hard drives, opticalstorage devices, etc. or any other appropriate computer-readable mediumor storage device). In one embodiment, the computer-executableinstructions may include lines of compiled C++, Java, hypertext markuplanguage (HTML), or any other programming or scripting code.

Additionally, the functions of the disclosed embodiments may beshared/distributed among two or more computers in or across a network.Communications between computers implementing embodiments can beaccomplished using any electronic, optical, radio frequency signals, orother suitable methods and tools of communication in compliance withknown network protocols.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,product, article, or apparatus that comprises a list of elements is notnecessarily limited only those elements but may include other elementsnot expressly listed or inherent to such process, product, article, orapparatus. Further, unless expressly stated to the contrary, “or” refersto an inclusive or and not to an exclusive or. For example, a conditionA or B is satisfied by any one of the following: A is true (or present)and B is false (or not present), A is false (or not present) and B istrue (or present), and both A and B are true (or present).

Additionally, any examples or illustrations given herein are not to beregarded in any way as restrictions on, limits to, or expressdefinitions of, any term or terms with which they are utilized. Instead,these examples or illustrations are to be regarded as being describedwith respect to one particular embodiment and as illustrative only.Those of ordinary skill in the art will appreciate that any term orterms with which these examples or illustrations are utilized willencompass other embodiments which may or may not be given therewith orelsewhere in the specification and all such embodiments are intended tobe included within the scope of that term or terms. Language designatingsuch nonlimiting examples and illustrations includes, but is not limitedto: “for example,” “for instance,” “e.g.,” “in one embodiment.”

Benefits, other advantages, and solutions to problems have beendescribed above with regard to specific embodiments. However, thebenefits, advantages, solutions to problems, and any component(s) thatmay cause any benefit, advantage, or solution to occur or become morepronounced are not to be construed as a critical, required, or essentialfeature or component.

What is claimed is:
 1. A system, comprising: a processor; a data store,the data store storing: spot viewing data comprising data on user viewsof a first spot associated with an entity through an offline network,and user interaction event data comprising data on user interactionswith an online presence on an online network, wherein the onlinepresence is associated with the entity associated with the first spot;and a non-transitory computer readable medium comprising instructionsfor: associating one or more user views of the first spot through theoffline network with corresponding user interactions with the onlinepresence of the entity on the online network; determining a first set ofuser views of the first spot that are associated with corresponding userinteractions with the online presence on the online network; determininga first cumulative probability curve for the first set of user viewswhich accounts for users who have responded incrementally to the firstspot airing on the offline network; determining a second cumulativeprobability curve corresponding to user interactions with the onlinepresence on the online network associated with users that did not viewthe first spot airing on the offline network; determining a latencyfactor to account for the effects of the first spot that occur outside abaseline window based on the difference between the first and secondcumulative probability curves, wherein the latency factor reflects arelationship between an expected total incremental response at theentity's online presence on the online network relative to an immediateincremental response, wherein the expected total incremental responseincludes both a response to the spot airing on the offline networkinside and outside the base time window; adjusting a metric associatedwith the online presence based on the latency factor; and generating aninterface based on the adjusted metric.
 2. The system of claim 1,wherein, the spot viewing data comprises data on user views of a secondspot, and the instructions of the non-transitory computer readablemedium comprising instructions are for: associating one or more userviews of the second spot through the offline network with correspondinguser interactions with the online presence of the entity on the onlinenetwork, wherein the second spot aired at a similar time to the firstspot; determining a second set of user views of the second spot that areassociated with corresponding user interactions with the online presenceon the online network; determining a first conditional probability curvefor the first set of user views; and determining a second conditionalprobability curve for the second set of user views; wherein the firstcumulative probability curve is determined based on the firstconditional probability curve for the first set of user views and thesecond conditional probability curve for the second set of user views.3. The system of claim 1, wherein the data store stores spot airing datacomprising data on when the spot associated with the entity was aired onthe offline network, and the instructions are for enhancing the spotairing data by adjusting a scheduled start time of at least one spotairing of the spot airing data.
 4. The system of claim 3, whereinenhancing the spot viewing data by associating each view of the spot inthe spot viewing data with a corresponding instance of the airing of thespot in the spot airing data based on time to adjust a spot viewing timeof the view of the spot.
 5. The system of claim 4, wherein theassociation is done using an IP address associated with both a user viewof the spot and the corresponding user interaction with the onlinepresence.
 6. The system of claim 1, wherein the metric is lift orconversion rate.
 7. A method, comprising: accessing, by a computer, adata store storing spot viewing data and user interaction event data,the spot viewing data comprising data on user views of a first spotassociated with an entity through an offline network, and the userinteraction event data comprising data on user interactions with anonline presence on an online network, wherein the online presence isassociated with the entity associated with the first spot; associating,by the computer, one or more user views of the first spot through theoffline network with corresponding user interactions with the onlinepresence of the entity on the online network; determining, by thecomputer, a first set of user views of the first spot that areassociated with corresponding user interactions with the online presenceon the online network; determining, by the computer, a first cumulativeprobability curve for the first set of user views which accounts forusers who have responded incrementally to the first spot airing on theoffline network; determining, by the computer, a second cumulativeprobability curve corresponding to user interactions with the onlinepresence on the online network associated with users that did not viewthe first spot airing on the offline network; determining, by thecomputer, a latency factor to account for the effects of the first spotthat occur outside a baseline window based on the difference between thefirst and second cumulative probability curves, wherein the latencyfactor reflects a relationship between an expected total incrementalresponse at the entity's online presence on the online network relativeto an immediate incremental response, wherein the expected totalincremental response includes both a response to the spot airing on theoffline network inside and outside the base time window; adjusting, bythe computer, a metric associated with the online presence based on thelatency factor; and generating, by the computer, an interface based onthe adjusted metric.
 8. The method of claim 7, wherein, the spot viewingdata comprises data on user views of a second spot, and the instructionsof the non-transitory computer readable medium comprising instructionsare further translatable by the processor for: associating one or moreuser views of the second spot through the offline network withcorresponding user interactions with the online presence of the entityon the online network, wherein the second spot aired at a similar timeto the first spot; determining a second set of user views of the secondspot that are associated with corresponding user interactions with theonline presence on the online network; determining a first conditionalprobability curve for the first set of user views; and determining asecond conditional probability curve for the second set of user views;wherein the first cumulative probability curve is determined based onthe first conditional probability curve for the first set of user viewsand the second conditional probability curve for the second set of userviews.
 9. The method of claim 7, wherein the data store stores spotairing data comprising data on when the spot associated with the entitywas aired on the offline network, and the instructions are for enhancingthe spot airing data by adjusting a scheduled start time of at least onespot airing of the spot airing data.
 10. The method of claim 9, whereinenhancing the spot viewing data by associating each view of the spot inthe spot viewing data with a corresponding instance of the airing of thespot in the spot airing data based on time to adjust a spot viewing timeof the view of the spot.
 11. The method of claim 10, wherein theassociation is done using an IP address associated with both a user viewof the spot and the corresponding user interaction with the onlinepresence.
 12. The method of claim 7, wherein the metric is lift orconversion rate.
 13. A non-transitory computer readable medium,comprising instructions translatable by a processor for: accessing spotviewing data comprising data on user views of a first spot associatedwith an entity through an offline network, and user interaction eventdata comprising data on user interactions with an online presence on anonline network, wherein the online presence on the online network isassociated with the entity associated with the first spot through theoffline network; associating one or more user views of the first spotthrough the offline network with corresponding user interactions withthe online presence of the entity on the online network; determining afirst set of user views of the first spot that are associated withcorresponding user interactions with the online presence on the onlinenetwork; determining a first cumulative probability curve for the firstset of user views which accounts for users who have respondedincrementally to the first spot airing on the offline network;determining a second cumulative probability curve corresponding to userinteractions with the online presence on the online network associatedwith users that did not view the first spot airing on the offlinenetwork; determining a latency factor to account for the effects of thefirst spot that occur outside a baseline window based on the differencebetween the first and second cumulative probability curves, wherein thelatency factor reflects a relationship between an expected totalincremental response at the entity's online presence on the onlinenetwork relative to an immediate incremental response, wherein theexpected total incremental response includes both a response to the spotairing on the offline network inside and outside the base time window;adjusting a metric associated with the online presence based on thelatency factor; and generating an interface based on the adjustedmetric.
 14. The non-transitory computer readable medium of claim 13,wherein, the spot viewing data comprises data on user views of a secondspot, and the instructions of the non-transitory computer readablemedium comprising instructions are for: associating one or more userviews of the second spot through the offline network with correspondinguser interactions with the online presence of the entity on the onlinenetwork, wherein the second spot aired at a similar time to the firstspot; determining a second set of user views of the second spot that areassociated with corresponding user interactions with the online presenceon the online network; determining a first conditional probability curvefor the first set of user views; and determining a second conditionalprobability curve for the second set of user views; wherein the firstcumulative probability curve is determined based on the firstconditional probability curve for the first set of user views and thesecond conditional probability curve for the second set of user views.15. The non-transitory computer readable medium of claim 13, wherein thedata store stores spot airing data comprising data on when the spotassociated with the entity was aired on the offline network, and theinstructions are for enhancing the spot airing data by adjusting ascheduled start time of at least one spot airing of the spot airingdata.
 16. The non-transitory computer readable medium of claim 15,wherein enhancing the spot viewing data by associating each view of thespot in the spot viewing data with a corresponding instance of theairing of the spot in the spot airing data based on time to adjust aspot viewing time of the view of the spot.
 17. The non-transitorycomputer readable medium of claim 16, wherein the association is doneusing an IP address associated with both a user view of the spot and thecorresponding user interaction with the online presence.
 18. Thenon-transitory computer readable medium of claim 13, wherein the metricis lift or conversion rate.